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ABSTRACT 



Careful definition and measurement of the activities 
and outcomes cf higher education can contribute significantly to the 
resolution of some of the most vital issues related to the 
educational enterprise. The inventory of educational outcomes and 
activities presented in this report is divided into 4 sections: (1) 

instructional outcomes, including (a) cognitive attributes of 
students, such as general knowledge, competence in fields of special 
interest, basic and second language art skills, critical thinking and 
creativity; (b) affective attributes of students, such as attitudes, 
values and perceptions in the political, racial and ethical areas: 
and (c) tangible attributes of students, such as awards, health, 
gradepoint average, and affiliation with groups; (2) instructional 
environment variables, including (a) academic environment attributes, 
such as attrition and retention, faculty availability to students, 
and student stress; and (b) social environment attributes, such as 
racial and cultural mix, family attitude characteristics, and 
marriage rate; (3) research outcomes, such as research topics, new 
ideas and new inventions; and (4) public service outcomes, such as 
student and faculty involvement in the community, and cultural and 
recreational activities. For each of these variables, a 
definition/comment is provided and a suggested measurement approach 
is indicated. (AF) 
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The Western Interstate Commission for Higher Education 
(WICHE) is a public agency through which the 13 western 
states work together 

... to increase educational opportunities for westerners. 
... to expand the supply of specialized manpower in the 
West. 

... to help universities and colleges improve both their 
programs and their management. 

... to inform the public about the needs of higher educa- 
tion. 



Executive Director: WICHE 
Robert H. Kroepsch 

Associate Director, WICHE, and 
Director, Planning and Management 
Systems Division: 

Ben Lawrence 



Director, Research Program: 

Robert A. Wallhaus 

Director, Development and Applications 
Program: 

Warren W. Gulko 



The program of the WICHE Planning and Management 
Systems Division was proposed by state coordinating agen- 
cies and colleges and universities in the West to be under 
the aegis of the Western Interstate Commission for Higher 
Education. The Planning and Management Systems Di- 
vision program proposes in summary: 



Director, Training Program: 
Robert Huff 



Program Associate: 
Charles R. Thomas 



To design, develop, and encourage the implementation of 
management information systems and data bases including 
common data elements in institutions and agencies of higher 
education that will: 

• provide improved information to higher education ad- 
ministration at all levels. 



Program Associate: 
John Minter 



• facilitate exchange of comparable data among institu- 
tions. 



# facilitate reporting of comparable information at the 
state and national levels. 



This publication is in the public domain in accordance 
with the Ford Foundation, grant number 700-0434. 
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THE PURPOSE OF THIS PAPER 



One of the initial major objectives of the WICHE Planning and 
Management Systems endeavor was the development of methods of measur- 
ing the outputs of educational programs and activities. After lengthy 
consideration of the output measurement problem, WICHE is making a 
start toward development of an inventory aimed at identifying the 
benefits of higher education and suggesting possible methodologies for 
measuring specific variables identified within the inventory. 

The material presented in this paper represents a first tentative 
step along an extremely difficult path. It is imperative that WICHE 
ascertain as quickly as possible whether the direction and approach 
initially chosen for this project are responsive to the needs of the 
higher education comnunity, as well as responsive to the public interest 
in accountability. 

Many institutions and education agencies, aware that WICHE is 
engaged in the development of output measures, have requested informa- 
tion regarding progress to date. This paper is intended to inform all 
interested persons about the nature and extent of WICHE ’s work in this 
area. At the same time, it will serve as an invitation to those inter- 
ested in higher education to forward their comnents and evaluations. 

All responses will be appreciated and carefully considered. 



Ben Lawrence 

Associate Director of Western 
Interstate Commission for 
Higher Education and 
Director of the Planning and 
Management Systems Division 
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INVENTORY OF EDUCATIONAL OUTCOMES AND ACTIVITIES 



Introduction 

Improved accountability is one of the most frequently discussed 
subjects in higher education today. As educational institutions seek 
larger amounts of funds to meet rising demands for educational services, 
those responsible for the allocation of public and private funds are 
prompted to ask pointed questions concerning the efficiency of edu- 
cational operations. Two questions associated with accountability 
inevitably arise: What are the true costs of educational programs and 
services? What are the outcomes and products that are produced by those 
programs and services? Virtually no one is against education. However, 
with the increase in crucial social and environmental problems, educa- 
tion is no longer viewed by everyone as having the highest possible 
priority for expenditure of ever increasing amounts of public and private 
funds . 

There is currently a substantial thrust toward the development of 
program budgeting techniques and procedures for determining the true 
cost of educational programs and services. To examine the costs of 
educational programs with little or no evidence available related to 
the outputs of those programs offers relatively little advantage to 
educational decision makers. Almost all programs are good. The question 
is, which set of programs is best and most beneficial in light of current 
priorities? If all suggested or requested educational programs cannot 
be funded, how does one determine where available dollars will be 



allocated without knowledge of the anticipated outputs of the 
competing programs? Educators, legislators and the public at large 
want to know precisely what they are buying and what they are likely 
to receive for their educational expenditures. This issue pervades 
much of the concern about the management of higher education. Careful 
definition and measurement of the activities and outcomes of higher 
education can contribute significantly to the resolution of some of 
the most vital current issues related to the educational enterprise. 

Understandably, there is a certain amount of hesitancy on the part 
of many people within the educational community to engage in attempts 
to measure the outcomes of the educational programs they conduct. Can 
the difficult task of measuring educational outcomes be done with 
sufficient precision, reliability and validity to offer a true picture 
of the benefits of higher education? Many fear that it cannot. There 
is a human tendency to fear new information, especially when one is 
concerned about the completeness and validity of that information. 
Inevitably, measurement of outcomes will make some individuals and 
programs appear more productive and beneficial than others. It seems 
clear, however, that educators in an energetic and dynamic system should 
not tolerate the lack of close scrutiny of the educational operation. 

It is only through careful examination of outcomes that direction can 
be given to efforts to improve performance and more efficiently pursue 
objectives. Outstanding educational programs should be recognized while 
programs with low performance should likewise be identified. It seems 
impossible to justify an attitude which would exempt educational 
programs and services from careful examination and evaluation. 



Decision making in an environment of limited resources 
and virtually unlimited demands requires cost-benefit analysis. Some 
sort of cost-benefit analysis has always been done by legislators, 
boards of regents and administrators and will continue to be used in the 
future. The problem is that the cost-benefit analysis of the past has 
been largely subjective and intuitive. Nonetheless, judgments have 
been made regarding program priorities and benefits. Those who would 
continue to deprive decision makers of information and formal evalua- 
tions related to educational outcomes are essentially arguing for the 
continuation of a potentially misleading kind of cost-benefit analysis 
and decision making. As compared with having no information concerning 
outputs of educational programs, even crude approximations having 
reasonable validity would be highly beneficial. 

No one who has examined the task of evaluating higher education 
outcomes can avoid the conclusion that such evaluation is an extremely 
difficult undertaking. However, many voices citing the need for under- 
taking the outcome evaluation task are currently being heard. Dr, 
Frederick Balderston recently wrote: 

— we have bumped hard into the question of output and its 
measurement because, among other things, we are seeking now 
to link the resources used to the results achieved— in other 
words, to link inputs with outputs. It turns out that in 
the long history of concern about the processes and activities 
of education, we have achieved only a very imperfect grasp of 
the nature of its results. We are having to tackle the 
problems of output definition and measurement under forced 
draft, for higher education has come to the front of public 
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attention both as a major social problem and as a major 
contributor to social change and economic development.— The 
job we have to do is urgent, important and controversial. 

If we had time, we might do well to sympathize with ourselves 
for taking it on. 1 

Undoubtedly, initial attempts to evaluate educational outcomes 
will be crude and justifiably subject to some suspicion. On the 
other hand, if wp continue to be intimidated by the technical dif- 
ficulties and political resistance surrounding the outcome evaluation 
task, we can only continue to suffer increasing public criticism and 
reaction for our failure to engage in self-evaluation activity. 
Hopefully, we shall learn from the early attempts at evaluation and 
will be able to improve our measurement techniques over time in order 
to gain valid information concerning at least some outputs of higher 
education activities. 

We currently hear a great deal about long-range planning 
and its importance to efficient operation of educational organizations 
Management information systems which will help institutions forecast 
the resource requirements for various kinds and sizes of educational 
programs are currently being developed and implemented on a widespread 
basis. Output measures are needed in order to complement the program 
budgeting aspects of these management information systems. 



^Frederick E. Balderston, "Thinking About the Outputs of Higher 
Education," Ford Foundation Program for Research Program in University 
Administration, Paper P-5, University of California, May, 1970, Page 2 
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In the future, educational planners hope to state long^-inge 
objectives in operational terms and make use of adequate information 
systems in guiding short-range activities toward the most efficient 
and effective pursuit of the long-range goals. Management of higher 
education has long been an art. That art has involved dealing with 
people, developing a spirit of cooperation, and coordinating the 
efforts of many professionals so that the total thrust of the institu- 
tion is in the direction which is perceived most desirable. The art 
of administration will continue to be practiced. However, management 
information systems which produce objective data on both the costs of 
programs and the outcomes of programs, will add an element of manage- 
ment science to this administrative art. 

Evaluation Approach 

The first requirement of any approach to evaluating the outcomes 
of educational programs is that the methodology be both practical and 
possible. It would be fairly easy to identify many measures for 
variables related to students, instruction, research, public services, 
etc., which would be highly desirable for evaluation purposes but 
would be virtually impossible to obtain. A pragmatic approach dictates 
that the measurement attempts be in the realm of the practicable 
rather than the impossible. 

If an array or inventory of outcome and activity measures related 
to educational programs is displayed, no one measure will assume a 
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dominant position. The WICHE approach to output measurement 
is to construct a profile or inventory of educational outcomes and 
activities. Viewing the total inventory, one should obtain not 
only new information concerning discrete variables of individual 
concern, but also develop a general understanding of the overall 
instructional effectiveness, academic environment and social contri- 
butions of the institution. 

In constructing a comprehensive list of variables and concerns 
it is inevitable that many variables will be identified for which no 
measurement technique can be immediately developed. This does not 
mean that such currently unmeasurable variables should be ignored or 
left off the list. Those who view an inventory of educational outcomes 
should be made fully aware of the areas which have not been evaluated 
as well as the measures which have been derived. It appears likely that 
many of the most important areas of concern for particular constitu- 
encies will be unmeasurable during the initial stages of developing an 
outcome inventory. By presenting a comprehensive list of variables 
with measures for only the variables it is currently possible to 
evaluate objectively, those who examine the inventory of outcomes 
will be made fully aware of the shortcomings, gaps and omissions in 
the list of outcome measures presented. Also, it is hoped that 
the lack of evaluation techniques for certain highly important 
variables will stimulate researchers to investigate and develop new 
measurement techniques for such variables. 



For intrainstitutional use, an inventory of educational outcome 
measures should be viewed as essentially neutral information. Each 
individual who examines such a list of measures will apply his own 
value system and establish in his own mind the relative importance 
of various items on the list. Different kinds and sizes of institu- 
tions have different objectives and should properly place different 
levels of importance on various outcomes and activities. A result 
which one institution considers highly successful might be considered 
only mediocre by another, depending upon the objectives and clientele 
of the respective institutions. 

The cost of gathering information is always a legitimate concern. 
Many institutions may wish to concentrate their efforts on particular 
areas which are of greatest concern in their unique settings. It is 
highly unlikely that all institutions would choose to measure all 
variables on a suggested inventory list. By viewing the entire list 
and selecting those variables with the greatest relevance to their 
institutional objectives, each educational organization can construct 
at reasonable cost an output accounting structure which is most 
appropriate for its own decision makers and clientele. 

Two basic criteria were used in selecting the variables for the 
suggested inventory. First, the inventory should be designed to be 
of service to many different kinds and levels of decision makers. 

Such decision makers include students, parents and faculty as well as 
administrators, regents and legislators. Therefore, a variable was 
included if it was felt to be of value to one or more of these groups. 



Students need more pertinent information in order to make better 
decisions regarding the selection of institutions which will provide 
the most appropriate educational experience for them. Faculty con- 
templating association with a particular institution need information 
concerning its academic and political environment. Individuals respon- 
sible for allocation and utilization of educational resources need to 
account for those resources in terms of benefits derived and goals 
achieved. These and many like considerations were influential during 
the development of the list of variables on the WICKE Inventory of 
Educational Outcomes and Activities. 

A second criterion used during construction of the list of 
inventory variables provides that a relatively complete characterization 
of the institution and its activities be furnished. Thus, the variables 
are intended to cover not only the academic and instructional aspects 
of institutional operation but also to reveal the social and economic 
impact and the benefits of research and public service activities. 

Examination of the inventory (attached herewith) will disclose 
that it is divided into four sections as follows: 

Section 1 : Instructional Outcomes 

A. Cognitive Attributes of Students 

B. Affective Attributes of Students 

C. Tangible Attributes to Students 
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Section 2: Instructional Environment Variables 

A. Academic Environment Attributes 



B. Social Environment Attributes 

Section 3i Research Outcomes 

Section 4: Public Service Outcomes 

The initial list of variables displayed in this paper relates 
primarily to the current status of institutions and the character- 
istics of students during their college years. Clearly, many of 
the most important contributions of higher education to the larger 
society stem from the productivity, participation and contributions 
of educated people some years after their college careers are 
completed. Follow-on studies which measure such later contributions 
and characteristics of students must eventually become an integral 
part of a complete output accounting structure. The focus during 
this initial stage, however, is on data related to students at times 
of entrance and exit from the institutions. 

The current organization of the inventory is purely a matter of 
convenience and represents the thoughts of those who have worked on 
the project. One could easily rearrange the list of variables to 
reflect the priorities of different institutions. However, the 
WICHE format of the inventory should serve as a beginning reference 
point for all users. 

The inventory presented within this paper contains three kinds 
of information. First, a title for each variable on the list is 
provided. Also, a definition and/or comment concerning the variable 



is provided and for those variables for which it seems currently 
feasible, a suggested measurement approach is indicated. As previously 
mentioned, many variables have been placed on the list because of their 
perceived importance to comprehensive characterization of the institu- 
tions, even though no viable measurement technique can currently be 
identified. It is hoped that future research and investigation will 
help eliminate some of these measurement gaps. 

The sources of suggested measures for the variables listed include 
test scores, data collected from student, faculty and other records, and 
responses to questionnaires. Recommendations for the exact procedures 
for administration of tests and questionnaires are yet to be determined. 
It also should be fully understood that the lists of variables and 
suggested measurement techniques are highly tentative and can be expected 
to change considerably as practitioners review them and researchers 
develop additional measures. 

The development of test instruments and questionnaires is not an 
appropriate area of endeavor for WICHE staff. Rather, those organi- 
zations with long experience and established expertise in the testing 
field should engage in the development of such instruments and question- 
naires as are required to support the implementation of the WICHE- 
defined evaluation approach. Well constructed, validated questionnaires 
and test instruments, which provide a convenient means of measuring 
specific variables among those identified within the WICHE inventory. 





must be made available if outcome evaluation efforts are to become 
widespread. WICHE, in cooperation with other organizations, will 
expend considerable energy on the identification of an acceptable evalua- 
tion methodology and the list of variables. Other organizations are 
expected to provide the measurement instruments and concomitant 
scoring and tabulating services to educational institutions. 

Implementation 

Implementation of measurement techniques to evaluate the outcomes 
of educational programs and activities will require the commitment of 
time, effort and money by institutions. Many individuals have suggested 
that such commitment of resources is well worth the anticipated conse- 
quence of having better information concerning the outcomes of higher 
education programs. Many questions will inevitably arise during 
initiation of such an evaluation effort. What is the best timing 
for submitting student and faculty questionnaires? How can student 
cooperation be gained in supplying carefully considered responses? 

What kind and volume of feedback is essential for questionnaire 
respondents? What sampling techniques should be employed? How will 
the evaluations be displayed and what are the aggregates of students 
whose characteristics and responses will be examined? 

Some standardization of methodology must be employed from year 
to year if institutions are to gather historical data which will allow 
comparability and analysis over time. Also, standardization of 
methodology will be necessary in order for any interinstitutional 
exchange of evaluation data to occur. These and many other problems 
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must be dealt with in the future if effective output measurement 
programs are to be established on local campuses. The initial 
Inventory of Educational Outcomes and Activities is intended primarily 
to stimulate widespread dialogue. Undoubtedly, many critical issues 
are yet to be identified. With review by all segments of the 
educational community and with direction from task forces composed 
of students and institutional personnel, the current inventory is sure 
to be altered substantially. From this first effort, our only hope 
is to learn by placing ourselves in the precarious position of 
suggesting specific measurement approaches and listening carefully 
to the reactions which are generated. This first effort will establish 
a point of departure for improved efforts in the future. 

Review Process 

No standardized approach to the measurement of educational outputs 
can be accomplished without reasonable consensus among all those concerned 
with higher education. Therefore, WICHE urges careful examination of 
the attached Inventory of Educational Outcomes and Activities. Letters 
of comment concerning the adequacy of the inventory or regarding the 
general approach suggested by the inventory are solicited from all 
interested persons. Each comment will be carefully considered. The 
review responses should provide valuable direction for WICHE 's future 
work in this area. 

Many individuals have been involved thus far in the development 
of WICHE' s approach to constructing an output accounting structure. In 
order to make a first step toward development of such a structure, WICHE, 
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the American Council on Education, and the Center for Research and 

Development in Higher Education at Berkeley conducted a National 

Invitational Seminar on the "Outputs of Higher Education", May 3-5, 

1970, in Washington, D. C. Papers written by knowledgable participants 

were distributed in advance of the seminar and each paper was discussed 

at length during the conference.* Contributors of major addresses and 

papers at the seminar included: 

Dr. Alexander Astin 
Office of Research 
American Council on Education 
Washington, D. C. 

Dr. Fred Balderston 

Vice President for Planning and Analysis 
University of California 
Berkeley, California 

Dr. John Bradl 

Director of the School of Public Affairs 
University of Minnesota 
Minneapolis, Minnesota 

Dr. David G. Brown 

Vice President for Academic Administration 
and Provost 
Drake University 
Des Moines, Iowa 

Dr. C. West Churchman 
Professor of Business Administration 
University of California 
Berkeley, California 

Dr. Alain Enthoven 
Vice President 
Litton Industries 
Beverly Hills, California 

Dr. Robbin Hough 
Professor of Economics 
Oakland University 
Rochester, Michigan 

*A publication of the seminar papers entitled. The Outputs of Higher 
Education : Their Identification , Measurement , and Evaluation , may be 
obtained by writing to WI CHE, P. 0. Drawer P, Boulder, Colorado, 80302. 

: i3 l7 




Dr. John Miller 
Dean of Graduate School 
Yale University 
New Haven, Connecticut 

Dr. Kenneth Tollett 
Canter for the Study of Democratic 
Institutions 

Santa Barbara, California 

Dr. John Vaizey 
Department of Economics 
Brunei University 
London, England 



Subsequent to the National Invitational Seminar a small committee 
of participants met in Boulder, Colorado, to organize the ideas generated 
in the conference. As a result of that work session, a tentative list 
of activity and output measures was constructed and has now served as 
the basis for the inventory presented in this paper. Additional review 
has been conducted with a group of students and representatives of 
various other segments of higher education. Following the national 
distribution of this paper, additional extensive review processes will 
be undertaken for each revision of the inventory. 
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*If more disaggregate data are 
desired, discipline specialties 
could be used. 
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Student Stress The degree of pressure and stress Tabulate the distribution cT 

experienced by students of various responses to appropriate student 
levels majoring in different questionnaire items for various 

fields. levels of student majors in each 

HEGIS Discipline Division. 
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